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DETAILED ACTION 
Response to A rgum ents 

1 . Applicant's arguments on page 11, third paragraph, filed July 26, 2004, with respect to 
claim 26 have been fully considered and are persuasive. The 35 U.S.C. 112, second paragraph 
rejection of claim 26 has been withdrawn. 

2. Applicant's arguments with respect to claims 1-32 have been considered but are moot in 
view of the new ground(s) of rejection. 

Claim Rejections - 35 USC §102 

3. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed pubUcation in this or a foreign country or in public use or on 
sale in this country, more than one year prior to the date of application for patent in the United States. 

4. Claims 1-4, 12-15, 23 and 24 are rejected under 35 U.S.C. 102(b) as being anticipated by 
Spight (USPN 4,462,046, previously cited). 

5. Regarding claim 1 , Spight discloses a robot system having an image processing function 
for determining orientation, or orientation and position of a robot operation on one of a plurality 
of objects (abstract), the system comprising: a robot 202 (Figure 2)\ a first image capturing 
device 40 capturing image data of the plurality of objects containing respective images of the 
objects (Figure 1; column 3, line 29-38; column 5, line 34-48; column 6, line 11-13); a memory 
storing reference models, each comprising an image of a reference object captured by said image 
capturing device in a different direction, and for each reference model storing information of the 
capturing direction of its associated image and information of an orientation of the robot with 
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respect to the reference object, the infonnation of the capturing direction representing a 
rotational posture (orientation) of the reference object relative to the robot, said reference object 
being one of the plurality of objects or an object having a shape identical to that of one of the 
plurality of objects (column 9, line 4-19); and a processor 64 to perform matching (correlation) 
on the image data containing images of the plurality of objects captured by said first image 
capturing device with each of said reference models successively to select one object having an 
image matched with one of said reference models (column 7, line 64-column 8, line 15; column 
9, line 59-column 10, line 1 1), and to determine orientation, or orientation and position of the 
robot operation based on the image of the selected one object, based on said one reference model 
and the information of its associated capturing direction, and based on the information of the 
orientation of the robot operation with respect to the reference object that is associated with said 
one reference model (column 7, line 28-49; column 8, line 16-37; column 11, line 21-37). 
6, Regarding claim 12, Spight discloses a robot system having an image processing function 
for determining orientation, or orientation and position of a robot operation on one of a plurality 
of objects of plural kinds, where the orientation of the operation corresponds to a determined 
orientation of the one object, where the determined orientation is a rotational posture of the one 
object (abstract), the system comprising: a robot 202 (Figure 2); a first image capturing device 40 
capturing image data of the plurality of objects containing respective images of the objects 
(Figure 1; column 3, line 29-38; column 5, line 34-48; column 6, line 1 1-13); a memory storing 
reference models, each comprising images of each of different kinds of reference objects 
corresponding to images captured by said image capturing device (column 8, line 10-15), and 
storing indicia of the kinds respectively associated with said reference models, and information 
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of a different orientation of the robot with respect to each of the different images of the reference 
object of each kind, where the captured information of orientation comprises information of a 
rotational posture of the reference object relative to the robot, each of said reference objects 
being one of the kinds of the plurality of objects or having a shape identical thereto (column 9, 
line 4-19); and a processor 64 to perform matching (correlation) on the image data containing 
images of the plurality of objects captured by said first image capturing device with each of said 
reference models successfully to select on object having an image matched with one of said 
kinds of the reference models (column 7, line 64-column 8, line 15; column 9, line 59-column 
10, line 11), and to determine orientation, or orientation and position of the robot operation, the 
determining based on the image of the selected one object, based on said one reference model, 
based on the indicia of the kind associated with said one reference model and the information of 
the orientation of the robot operation with respect to the reference object associated with said one 
reference model of said one kind (colunnn 7, line 28-49; column 8, line 16-37; column 1 1, line 
21-37). 

7. Regarding claims 2 and 13, Spight discloses that said reference models are obtained from 
a part of the image data of the reference object (column 3, line 29-38; column 6, line 37-52; 
column 10, line 50-63). 

8. Regarding claims 3 and 14, Spight discloses that said reference models are obtained by 
processing the image data of the reference object (column 3, line 29-38; column 6, line 37-52; 
column 10, line 50-63). 

9. Regarding claims 4 and 15, Spight discloses that said first image capturing device 40 
comprises a camera for capturing two-dimensional image data (column 5, hne 42-48). 
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10. Regarding claim 23, Spight discloses a method for automatically determining an 
arrangement of a workpiece relative to a robot, where the determined arrangement comprises at 
least rotational posture arrangement (orientation) of the workpiece relative to the robot (abstract), 
the method comprising: storing reference images corresponding to images of the workpiece or an 
object so shaped (workpiece/object) and reference arrangement information indicating 
arrangements (configurations) of the robot and workpiece/object relative to each other when the 
images were captured, the reference arrangement comprising rotational arrangements 
(orientations) of the workpiece relative to the robot (column 9, line 4-19); from a known 
arrangement of the robot, capturing a working image of the workpiece among a plurality of 
randomly arranged workpieces with an imaging device (column 3, line 29-38; column 6, Hne 1 1- 
13); finding one of the reference images matches (correlates with) the workpiece in the working 
image (column 7, line 64-column 8, line 15; column 9, line 59-column 10, line 1 1); and 
determining an arrangement of the robot relative to the workpiece based on information 
indicating the known arrangement of the robot, and based on the reference arrangement 
information corresponding to the found reference image, where the determined arrangement 
comprises rotational arrangement (orientation) of the workpiece relative to the robot (column 7, 
line 28-49; column 8, line 16-37; column 11, line 21-37). 

1 1 . Regarding claim 24, Spight discloses that reference images and reference arrangement 
information is obtained for workpieces/objects of different shapes (plurality of configurations of 
each desired object to be identified), and wherein the finding comprises first determining that a 
reference image of one of the different shapes matches the working image of the workpiece, and 
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then finding one reference image of the shape that best matches the working image (column 8, 
line 10-30; column 9, line 4-19). 

Claim Rejections - 35 USC §103 

12. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 

obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

13. Claims 5 and 16 are rejected under 35 U.S.C. 103(a) as being unpatentable over Spight as 
applied to claims 4 and 15 above, and further in view of Suyama et al. (USPN 4,879,664, 
previously cited, hereafter Suyama) or Stauffer (USPN 4,410,804, previously cited). 

14. Spight does not disclose that the image data are captured from a predetermined distance. 
Suyama discloses a three-dimensional position sensor comprising robot-teaching apparatus 
wherein said image data of the reference object are captured by said camera (Figure 1 la, element 
35) from a predetermined distance (column 6, line 20-37). In addition, Stauffer teaches that if a 
two-dimensional sensor is used, the image processor is unable to simultaneously determine the 
distance to the object unless the objects are always positioned at a known distance (column 1, 
line 3 1-39). It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to capture an image at a predetermined distance in order to adjust the 
position and posture of teaching head 3 with respect to the workpiece 9 as taught by Suyama 
(column 6, line 20-37) or such that the distance to the object need not be determined by other 
means as taught by Stauffer (column 1, hne 31-39). 
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15. Claims 6, 7, 1 1, 17, 18 and 22 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Spight as applied to claims 1 and 12 above, and further in view of Maeno et al. (USPN 
5,047,714, previously cited, hereafter Maeno). 

16. Regarding claims 6 and 17, Spight does not disclose that the robot moves a second image 
capture device to have the determined orientation and/or position. Maeno discloses a method of 
recognizing surface-mounted parts including a robot system comprising a second image 
capturing device (Figure 7c, element 10); wherein said robot situates said second image data 
capturing device to have said determined orientation or to have said determined orientation and 
said determined position with respect to the selected one object (column 4, line 44-49), and a 
processor processes second image data captured by said second image capturing device to detect 
position and/or rotational posture of the selected one object with respect to said second image 
data capturing device (column 4, line 54-63). It would have been obvious to one of ordinary 
skill in the art at the time the invention was made to move the second image capture device 
(imaging tube) to have the determined orientation and/or position as taught by Maeno in order to 
locate position detecting patterns located at specific positions to accurately and directly 
determine the position and direction of conductor patterns on the printed circuit board (column 1 , 
line 51-59). 

17. Regarding claims 7 and 18, Spight does not disclose that the robot moves a second image 
capture device to have the determined orientation and/or position. Maeno discloses a robot 
system comprising a second image capturing device for obtaining three-dimensional position; 
wherein said robot situates said second image data capturing device to have said determined 
orientation or to have said determined orientation and said determined position with respect to 
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the selected one object, so that the second image data capturing device is directed to a 
characterizing portion of the object; and wherein said processor detects the three-dimensional 
position and/or posture of the selected one object based on the position of said characterizing 
portion obtained by said second image data capturing device (see discussion of claims 6 and 17 
above). Maeno does not explicitly disclose that the three-dimensional position of the selected 
one object, but the examiner takes Official Notice that obtaining three-dimensional position of an 
object is well known in the art. It would have been obvious to one of ordinary skill in the art at 
the time the invention was made to detect the three-dimensional position of the object in order to 
determine the location of the object in space so that operations may be carried out upon it. 

18. Regarding claims 1 1 and 22, Spight discloses that said robot operation is an operation of 
picking up the selected one object from the plurality of objects, some of which are overlapped 
(randomly arranged) with each other (column 6, line 11-13). 

19. Claims 8, 19, 27 and 30 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Spight in view of Maeno as applied to claims 6, 7, 17 and 18 above, and further in view of 
Soderberg (USPN 4,785,528, previously cited). 

20. Regarding claims 8, 19, 27, and 30, Maeno does not disclose that said first image data 
capturing device is used as said second image data capturing device. Soderberg discloses a 
robotic work positioning system including a camera 34 mounted on member 28 of robot 10 
which serves as both the first and second image data capturing device since it both determines 
the position of the work, and is moved to have the determined orientation and/or position (Figure 
1; column 2, line 53-column 3, line 1, line 44-50; column 5, line 34-41). It would have been 
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obvious to one of ordinary skill in the art at the time the invention was made to use said first 
image data capturing device as said second image data capturing device as taught by Soderberg 
in order to retrieve parts fi"om a conveyor using a vision system to select the desired part fi"om 
other parts and engage the part at precisely the correct location to be properly placed by the robot 
(column 5, line 60-column 6, line 6). 

21. Claims 25 and 26 are rejected under 35 U.S. C. 103(a) as being unpatentable over Spight 
as applied to claim 23 above, and further in view of Soderberg. 

22. Regarding claim 25, Spight does not disclose that the robot is used to capture the 
reference images. Soderberg discloses a robot that is used to capture reference images, and 
wherein the reference arrangement information represents arrangements of the robot when 
.capturing the reference images (Figure 1; column 2, line 53-colunm 3, line 4, line 31-59; column 
5, line 60-column 6, line 6). It would have been obvious to one of ordinary skill in the art at the 
time the invention was made to use a robot to capture reference images as taught by Soderberg in 
order to retrieve parts fi'om a conveyor using a vision system to select the desired part from other 
parts and engage the part at precisely the correct location to be properly placed by the robot 
(column 5, line 60-column 6, line 6). 

23. Regarding claim 26, Spight does not disclose that a second imaging device is affixed to 
the robot. Soderberg discloses a camera 34 mounted on member 28 of robot 10 and is used to 
determine additional arrangement information used to determine the known arrangement of the 
robot relative to the workpiece (Figure 1; column 3, line 44-50; column 5, line 34-41). It would 
have been obvious to one of ordinary skill in the art at the time the invention was made to affix 
an imaging device to the arm of a robot as taught by Soderberg in order to retrieve parts from a 
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conveyor using a vision system to select the desired part from other parts and engage the part at 
precisely the correct location to be properly placed by the robot (column 5, line 60-column 6, 
line 6). 

24. Claims 9, 10, 20, 21, 28, 29, 31 and 32 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Spight in view of Maeno as applied to claims 6, 7, 17 and 18 above, and 
further in view of Sakakibara et al. (JP 07-270137, previously cited, hereafter Sakakibara). 

25. Regarding claims 9, 20, 28, and 3 1 , Maeno does not disclose that said second image 
capturing device comprises a three dimensional visual sensor of spot-Hght scanning type. 
Sakakibara discloses a three dimensional visual sensor usable in robot automation (page 1, 
paragraph 1), of spot-Hght scanning type capable of measuring distance between the sensor and 
an object (page 3, paragraph 15). It would have been obvious to one of ordinary skill in the art at 
the time the invention was made to utilize a three dimensional visual sensor of spot-light 
scanning type as taught by Sakakibara in order to quickly and accurately determine the three 
dimensional position of an object using one device (page 2, paragraph 1 1). 

26. Regarding claims 10, 21, 29, and 32, Sakakibara discloses an image data capturing device 
comprising a structured-light unit for irradiating a structured light on the selected object and 
capturing an image of the object including the irradiated light on the object (page 3, paragraph 
16). 

Conclusion 

27. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. 
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• USPN 4,416,924 to Birk et al. discloses a system for visually determining position 
and/or orientation in space. 

• USPN 4,437,1 14 to LaRussa discloses a robotic vision system including grasping 
objects with random orientation by finding a matching image. 

• USPN 4,704,694 to Czemiejewski discloses a learned part system including 
identifying the presence and orientation of an object to control a robotic system by 
comparing an unknown object with stored values. 

• USPN 4,707,647 to Coldren discloses a gray scale vision method and system 
including determining the location and orientation of an object. 

• USPN 4,942,539 to McGee et al. discloses a method and system for automatically 
determining the position and orientation of an object in 3-D space. 

• USPN 5,220,619 to Keokoek discloses a method of matching a variable 2-D image of 
a known 3-D object with a desired 2-D image of the object to automatically determine 
the position and orientation of the object in 3-D space. 

• USPN 5,329,469 to Watanabe discloses a calibration method for a visual sensor 
including determining the position and orientation of a workpiece in a robot 
coordinate system 

• USPN 5,621,807 to Eibert et al. discloses an intelligent range image camera for object 
measurement including comparing 3-D object geometry data with 3-D geometric 
models to identify an object as well as its position and orientation. 

• USPN 5,790,687 to McLaughlin et al. discloses a method and apparatus for the 
optical determination of the orientation of a garment workpiece. 
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• USPN 6,490,369 to Beiman discloses a method of viewing and identifying a part for a 
robot manipulator including a stored template of a known part. 
28. Applicant's amendment necessitated the new ground(s) of rejection presented in this 
Office action. Accordingly, TfflS ACTION IS MADE FINAL. See MPEP § 706.07(a). 
Applicant is reminded of the extension of time pohcy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS fi*om the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1 .136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Ryan J Hesseltine whose telephone number is 703-306-4069. 
The examiner can normally be reached on Monday - Friday, 8:30 AM - 5 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Amelia Au can be reached on 703-308-6604. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 
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Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished . 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 



Ryan J. Hesseltine 
January 10, 2005 



